Camera-Based Scanning

ABSTRACT

Embodiments of camera-based scanning are described. In various embodiments, one or more quadrangular objects are automatically selected from a captured image for scanning. The automatic selection is determined to be successful based on the selected quadrangular objects having an associated score that exceeds a predefined threshold. One or more scanned documents are created from portions of the captured image corresponding to the selected quadrangular objects, and the created scanned documents include corrections for perspective distortions of the selected quadrangular objects in the captured image.

RELATED APPLICATION

This application is a continuation of and claims priority to U.S. patentapplicant Ser. No. 12/578,445, filed on Oct. 13, 2009, which claims thebenefit of U.S. Provisional Application Ser. No. 61/245,278 filed Sep.23, 2009, the disclosures of which are incorporated by reference hereinin their entirety.

BACKGROUND

Computing devices that include cameras are increasingly more common andmobile, such as laptop computers, tablet PCs, digital camera devices,mobile phones, ultra-mobile PCs, as well as other mobile data,messaging, and/or communication devices. Users may make use of camerasassociated with computing devices to take various pictures includingcapturing images of presentations, whiteboards, business cards,documents, sketches, paintings, and so forth. The users may then be ableto refer to the captured images to recall information contained thereinsuch as diagrams, pictures, lists and other text, and so forth. Often,the users would like to be able to make use of information from capturedimages in their own documents, notes, and/or presentations.Traditionally, however, images captured by a camera are static andextracting electronically usable and/or editable information from thestatic images may not be straightforward.

A traditional technique utilized to generate a scanned version of acaptured image involves obtaining a print of the image and then manuallyoperating a scanner to create a scanned version of the image. Anothertraditional technique used to work with a captured image entailstransferring the image from a capture device to a desktop computer andthen using an image editing application of the desktop computer tofurther process the image. Such manually intensive techniquestraditionally employed to obtain information contained in capturedimages may be inconvenient and time consuming for the user.

SUMMARY

This summary is provided to introduce simplified concepts ofcamera-based scanning. The simplified concepts are further describedbelow in the Detailed Description. This summary is not intended toidentify essential features of the claimed subject matter, nor is itintended for use in determining the scope of the claimed subject matter.

Embodiments of camera-based scanning are described. In variousembodiments, scanned documents can be created using images captured by acamera associated with a device. An image captured by the camera isprocessed to identify quadrangular portions within the image thatcorrespond to rectangular objects such as paper, business cards,whiteboards, screens, and so forth. One or more of these quadrangularportions can be selected for scanning automatically based on a scoringscheme and/or semi-automatically with the aid of input from a user. Oneor more scanned documents are created from the selected quadrangularportions by un-warping the selected portions to remove effects ofperspective (e.g., rectify the portions to rectangles) and applyingvarious image enhancements to improve appearance.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of camera-based scanning are described with reference to thefollowing drawings. The same numbers are used throughout the drawings toreference like features and components:

FIG. 1 illustrates an example of a device that can implement variousembodiments of camera-based scanning.

FIG. 2 illustrates an example system in which embodiments ofcamera-based scanning can be implemented.

FIG. 3 illustrates example method(s) for camera-based scanning inaccordance with one or more embodiments.

FIG. 4 illustrates other example method(s) for camera-based scanning inaccordance with one or more embodiments.

FIG. 5 illustrates various components of an example device that canimplement embodiments of camera-based scanning.

DETAILED DESCRIPTION

Embodiments of camera-based scanning provide techniques for a user of asuitably configured computing device to scan documents, presentations,and other subjects using images captured by a camera associated with thedevice. Camera-based scanning can correct for the effects of perspectiveon rectangular objects such as paper, business cards, whiteboards,screens, and so forth.

For example, a user can aim the camera of a portable device towards asubject and initiate a capture of an image of the subject using abutton, touch, or other suitable input. When the capture is initiated bythe user, a capture operation can occur to capture an image of thesubject. The image capture can initiate various processing of thecaptured image to create one or more scanned documents from the capturedimage. In an embodiment, a touch input or touch event on a touch-screencan be initiated to indicate both an area of interest and that the imagecapture should occur. The location of the touch input can be used insubsequent processing steps to guide a scoring function. This processingcan include identifying quadrangles within the captured image to beconsidered as potential portions for scanning. The device may beconfigured to select one or more of the identified quadranglesautomatically based on a scoring scheme, and/or semi-automatically withthe aid of input from the user. One or more scanned documents can thenbe created by un-warping the selected quadrangles to remove effects ofperspective (e.g., rectify the quadrangles to rectangles) and byapplying various image enhancements to improve appearance.

While features and concepts of the described systems and methods forcamera-based scanning can be implemented in any number of differentenvironments, systems, and/or various configurations, embodiments ofcamera-based scanning are described in the context of the followingexample systems and environments.

FIG. 1 illustrates an example 100 of a computing device 102 that canimplement various embodiments of camera-based scanning. The computingdevice 102 is an example of various types of devices including exampleportable devices described with reference to FIG. 2 and can also beimplemented with any number and combination of differing components asdescribed with reference to the example device shown in FIG. 4. Thecomputing device 102 includes an integrated display screen 104 todisplay user interfaces, user interface elements and features,user-selectable controls, various displayable objects, and the like. Thecomputing device 102 also includes a camera 106 to capture digitalimages. In the depicted example, the camera 106 is shown on a side ofthe computing device 102 opposite from the display screen 104.

The computing device 102 further includes at least an input driver 108to process various inputs from a user to operate the computing device102. In at least some embodiments, the display screen 104 is atouch-screen and the input driver 108 is operable to detect and processa variety of touch inputs and/or touch events. In an embodiment, a touchinput or touch event on the touch-screen 104 can be initiated toindicate both an area of interest and to initiate an image capture. Animage can be displayed on the display screen as a preview of the imageto be captured, and a touch event at a particular location on the screenindicates that the image should be captured. Additionally, theparticular location is identified as being of interest to a scoringfunction in subsequent processing algorithms. Accordingly, a touch inputcan be utilized to select part of an image and to initiate the camerataking a picture to capture the image.

The computing device 102 also includes a capture application 110 toinitiate display of a user interface 112 and the various user interfaceelements, features, and controls to facilitate capturing images via thecamera 106 and processing of the images. Further, the captureapplication 110 represents functionality of the computing device 102 toimplement camera-based scanning techniques described herein. The exampleuser interface 112 is illustrated as a split-screen interface having aviewfinder 114 and a scanned image display 116. The viewfinder 114 canpresent current images from the camera 106 and switch to present acaptured image when a picture is taken. Further, a user may be able tomodify and select portions of a captured image via interaction with theviewfinder 114.

The scanned image display 116 can present one or more portions of thecaptured image that are processed by the capture application 110 toproduce scanned documents. The split-screen enables simultaneous displayof a captured image in the viewfinder 114 and a scanned documentproduced from the captured image in the scanned image display 116. Inthis manner, a user can see both a captured image and a scanned portionof the image, and can intuitively make adjustments using the userinterface 112, such as modifying boundaries of a selected portion orselecting a different portion.

In an embodiment of camera-based scanning, a user of a computing device102 can initiate camera-based scanning by taking a picture of a subject118. The subject 118 of the picture can include one or more rectangularobjects such as a document, paper, business card, photo, whiteboard, andso forth. In the example of FIG. 1, the subject 118 appears as a displayscreen that is being used for a business presentation. When a picture istaken to initiate camera-based scanning, capture application 110captures an image of the subject 118 and can output the image in theviewfinder 114 of the user interface 112.

Capture application 110 is implemented to detect portions of thecaptured image that correspond to rectangular objects. In particular,capture application 110 can be configured to identify quadrangles withinthe captured image to be considered as potential regions for scanning.Quadrangles within an image can be identified using a variety of featureextraction techniques suitable to find arbitrary shapes within imagesand other documents.

In at least some embodiments, the capture application 110 includes orotherwise makes use of an edge detector operable to detect edges basedupon visual differences, such as sharp changes in brightness. Oneexample algorithm suitable for edge detection is Canny's algorithm. Whenedges have been identified, the edges may be joined into connected linesto form quadrangles. For instance, vertices (corners) can be identifiedthrough the edge detection and then the vertices can be connected toform quadrangles. This can involve, for example, application of a linearHough transform to correct for imperfections in the detected edges andderive lines corresponding to the edges. Accordingly, a set of potentialquadrangles can be derived using the detected edges and lines, where thelines are detected from similarly-oriented edges along a particulardirection and are then combined to form the quadrangles.

Capture application 110 can cause indicators 120 to appear in the userinterface 112 to represent quadrangles that are detected and/or selectedwithin a captured image. For example, the captured image in theviewfinder 114 of FIG. 1 includes indicators 120 configured as points atvertices and dashed lines appearing along edges. Other indicators 120are also contemplated, such as animated lines, color changes, flags, andso forth. Potential quadrangles can be presented via the viewfinder toenable a user to select one or more of the quadrangles for scanning.

In an implementation, the capture application 110 can be configured toautomatically select one or more potential quadrangles to scan basedupon a scoring scheme. The scoring scheme can rate potential quadranglesbased upon a variety of criteria including for example size, location,recognition of content such as text and faces, and so forth. A highestscoring quadrangle can be selected automatically. In another example,any quadrangles that exceed a threshold score can be selected.

The capture application 110 can be further configured to implementsemi-automatic selection of quadrangles when automatic selection failsto find suitable quadrangles and/or when a user initiates semi-automaticselection. To do so, automatically selected quadrangles, corners, and/orlines may be presented in the user interface 112 using indicators 120 asnoted above. A user can then provide input to modify the automaticallyselected quadrangles, eliminate a quadrangle, define a customquadrangle, and so forth.

In an implementation, the indicators 120 may be selectable by the userto modify a quadrangle, such as by dragging of a corner to repositionthe corner. In another example, a user may define a custom quadrangle byselecting one corner through interaction with the user interface 112.The capture application 110 can be configured to automatically derive acorresponding quadrangle based upon the user selection of one corner.The user may also operate a select and drag tool of the user interface112 to identify an area for a custom quadrangle. When the computingdevice 102 is touch-capable, a user may touch and drag directly on thedisplay screen 104 to modify and define quadrangles. Other input devicesmay also be employed for semi-automatic selection of quadranglesincluding for example a stylus, mouse, direction keys, and/or othersuitable input devices.

One or more scanned documents can then be created by un-warping theselected quadrangles to remove effects of perspective (e.g., rectify thequadrangles to rectangles) and applying various image enhancements toimprove appearance. In particular, to perform the un-warping the captureapplication 110 can be implemented to detect and correct for distortionsdue to the perspective of a captured image. For instance, the captureapplication 110 can determine the perspective based upon angles andratios of a selected quadrangle. The capture application 110 can cropthe captured image to correspond to the selected quadrangle. The captureapplication 110 can then rotate, resize at least some portions, andotherwise make corrections to the cropped image to account for theperspective distortions and produce an un-warped image that is rectifiedto a rectangle.

Capture application 110 can also apply various image enhancements toimprove appearance of the un-warped image. Examples of these imageenhancements include color enhancements, corrections for lighting andshadows, and background removal to name a few. The image enhancementsmay also include applying optical character recognition (OCR) to theun-warped image to identify text and produce scanned documents havingeditable text portions.

Consider the particular example of the business presentation shown inFIG. 1. The business presentation appears on a display screen andincludes text and an illustration of a person. When a user takes apicture of the business presentation, an image is captured and canappear in the viewfinder 114. Note that the image appears skewed due tothe angle at which the picture is taken. Initially, capture application110 can identify the screen boundaries using camera-based scanningtechniques discussed herein. Accordingly, a quadrangle corresponding tothe screen is identified in the viewfinder 114 using indicators 120.Further, capture application 110 could initially present a scannedversion of the screen including the text and illustration via thescanned image display 116.

In the depicted example, however, the user has selected a customquadrangle to choose the illustration without the text. For instance,the user may touch the display screen 104 of the computing device 102 topick a corner and/or drag a selection box around the illustration.Responsive to this selection, the selected portion (e.g., theillustration) has been scanned by un-warping to remove effects ofperspective (e.g., the skew) and applying enhancements. The resultingscanned illustration appears in the scanned image display 116 and can bepresented simultaneously with the captured business presentation in theviewfinder 114 using the split-screen of the user interface 112. Theuser may make use of the scanned illustration in a variety of ways, suchas by adding annotations, sharing with colleagues, publishing on awebsite or blog, and so forth.

FIG. 2 illustrates an example system 200 in which various embodiments ofcamera-based scanning can be implemented. Example system 200 includes aportable device 202 (e.g., a wired and/or wireless device) that can beany one or combination of a mobile personal computer 204, a personaldigital assistant (PDA), a mobile phone 206 (e.g., cellular, VoIP, WiFi,etc.) that is implemented for data, messaging, and/or voicecommunications, a portable computer device 208 (e.g., a laptop computer,a laptop computer with a touch-screen, etc.), a media device 210 (e.g.,a personal media player, portable media player, etc.), a gaming device,an appliance device, an electronic device, and/or any other type ofportable device that can receive, display, and/or communicate data inany form of audio, video, and/or image data.

Each of the various portable devices can include an integrated displayand/or an integrated touch-screen or other display, as well asselectable input controls via which a user can input data and/orselections. For example, mobile personal computer 204 includes anintegrated touch-screen 212 on which a user interface 214 can bedisplayed that includes displayable objects and/or user interfaceelements 216, such as any type of image, graphic, text, selectablebutton, user-selectable control, menu selection, map element, and/or anyother type of user interface displayable feature or item. The userinterface 214 can also display captured and scanned images via asplit-screen in accordance with one or more embodiments of camera-basedscanning described herein.

Any of the various portable devices described herein can be implementedwith one or more sensors, processors, communication components, datainputs, memory components, storage media, processing and controlcircuits, and/or a content rendering system. Any of the portable devicescan also be implemented for communication via communication networksthat can include any type of a data network, voice network, broadcastnetwork, an IP-based network, and/or a wireless network that facilitatesdata, messaging, and/or voice communications. A portable device can alsobe implemented with any number and combination of differing componentsas described with reference to the example device shown in FIG. 4. Aportable device may also be associated with a user (i.e., a person)and/or an entity that operates the device such that a portable devicedescribes logical devices that include users, software, and/or acombination of devices.

In this example, portable device 202 includes one or more processors 218(e.g., any of microprocessors, controllers, and the like), acommunication interface 220 for data, messaging, and/or voicecommunications, and data inputs 222 to receive media content 224. Mediacontent (e.g., to include recorded media content) can include any typeof audio, video, and/or image data received from any media content ordata source, such as messages, television media content, music, videoclips, data feeds, interactive games, network-based applications, andany other content. Portable device 202 is implemented with a devicemanager 226 that includes any one or combination of a controlapplication, software application, signal processing and control module,code that is native to the particular device, and/or a hardwareabstraction layer for the particular device.

Portable device 202 includes various software and/or media applications228 that may incorporate components such as capture applications 230that can be processed or otherwise executed by the processors 218. Themedia applications 228 can include a music and/or video player, animaging application, a Web browser, an email application, a messagingapplication, a digital photo application, and the like. Portable device202 includes a rendering system 232 to render user interfaces from thecapture applications 230 to generate a display on any of the portabledevices. The rendering system 232 is also implemented to receive andrender any form of audio, video, and/or image data received from anymedia content and/or data source. Portable device 202 also includes acamera 234 and an input driver 236 that can incorporate or otherwisemake use of a touch-screen driver for the touch-screen 212. The inputdriver 236 can be configured to detect and process various inputs and/ordeterminable representations of gestures, inputs, and/or motions tooperate functionality of the portable device 202 including operation ofa capture application 230 to implement camera-based scanning.Implementations of a capture application 230 and the input driver 236are described with reference to the capture application 110 and inputdriver 108 shown in FIG. 1, and with reference to embodiments ofcamera-based scanning as described herein.

Example methods 300 and 400 are described with reference to respectiveFIGS. 3 and 4 in accordance with one or more embodiments of camera-basedscanning. Generally, any of the functions, methods, procedures,components, and modules described herein can be implemented usinghardware, software, firmware, fixed logic circuitry, manual processing,or any combination thereof. A software implementation represents programcode that performs specified tasks when executed by a computerprocessor. The example methods may be described in the general contextof computer-executable instructions, which can include software,applications, routines, programs, objects, components, data structures,procedures, modules, functions, and the like. The methods may also bepracticed in a distributed computing environment by processing devicesthat are linked through a communication network. In a distributedcomputing environment, computer-executable instructions may be locatedin both local and remote computer storage media and/or devices. Further,the features described herein are platform-independent and can beimplemented on a variety of computing platforms having a variety ofprocessors.

FIG. 3 illustrates example method(s) 300 of camera-based scanning. Theorder in which the method is described is not intended to be construedas a limitation, and any number of the described method blocks can becombined in any order to implement the method, or an alternate method.

At block 302, an input is detected that initiates capture of an image.For example, the input driver 108 at computing device 102 detects a userselection to take a picture of a subject 118. The subject 118 caninclude one or more rectangular objects that are detectable forcamera-based scanning. The capture may be initiated by user manipulationof a touch screen button, a key stroke, operation of a dedicated shutterbutton of the computing device 102, or another suitable input.

At block 304, an image is captured responsive to the input. For example,capture application 110 at computing device 102 can capture an image ofthe subject 118 using the camera 106 responsive to the input at block302. The captured image can be presented via a user interface 112 of thecomputing device 102.

At block 306, one or more portions of the captured image are scannedbased upon detection of quadrangles in the captured image. At block 308,enhancements are applied to the one or more scanned portions.Quadrangles can be detected in the captured image using a variety oftechniques. For example, capture application 110 at computing device 102can identify quadrangles using either or both of automatic andsemi-automatic techniques discussed in relation to FIG. 1. In anotherexample, a manual technique can be employed in which the captured imagecan be presented to a user via the computing device 102 to manuallyselect quadrangles. In this example, functionality to enable automaticdetection of quadrangles via the capture application 110 may bedisabled, may not be included, or may be otherwise unavailable. Variousenhancements to improve appearance of the one or more scanned portionsare contemplated. Further details regarding techniques to detectquadrangles and enhance scanned images are provided below with referenceto example method(s) 400 shown in FIG. 4.

At block 310, scanned documents that correspond to one or more portionsare output. For example, the capture application 110 at computing device102 can cause a scanned document to be presented in a scanned documentdisplay of a user interface 112. A user may then be able to work withthe scanned document such as by saving the document, adding annotations,sending the document to one or more recipients, and so forth.

FIG. 4 illustrates example method(s) 400 of camera-based scanning. Inparticular, FIG. 4 represents an example algorithm that is suitable toscan images captured via a camera 106 of a computing device 102. Theorder in which the method is described is not intended to be construedas a limitation, and any number of the described method blocks can becombined in any order to implement the method, or an alternate method.

At block 402, a captured image is pre-processed. For example, captureapplication 110 of a computing device 102 can pre-process capturedimages in various ways to prepare the images for camera-based scanning.By way of example, pre-processing can include applying image filters,enhancing contrast, adjustments to brightness, down scaling, GSconversion, median filtering, and so forth. In an embodiment,pre-processing includes one or more of down-scaling the image, contrastenhancement, and noise filtering. The down-scaling can occur to reduce aresolution of the image and the number of pixels to process for theimage. As resolution in an image is increased, more computing resourcesare consumed to process the image and more false (e.g., unwanted) edgescan result from edge detection. Accordingly, down-scaling can speed upprocessing and enable improved edge detection.

Contrast enhancement can be employed to set the lightest tones in theimage to white and the darkest tones to black. This can also improvedetection of edges and lines through detection algorithms that findsharp differences in contrast and/or brightness. Noise filteringinvolves applying one or more filters to remove image noise. Some noisefilters, such as Gaussian blur, can degrade (e.g., soften) edges ofimages, and make it more difficult to detect the edges. Accordingly,noise filtering techniques that preserve edges, such as bilateral and/ormedian filtering, can be employed with camera-based scanning techniquesto prevent edge degradation.

Following pre-processing, method 400 can proceed to perform detection ofquadrangles in the captured image. At block 404, edges are detected inthe captured image. Edges that are detected can correspond to borders ofrectangular objects within an image such as edges of a business card, aframe of a painting, edges of a display screen, and so forth. The edgescan be detected in any suitable way. For example, capture application110 of a computing device 102 can be implemented to apply Canny'salgorithm to detect edges based on sharp changes in contrast.

Canny's algorithm can make use of a configurable threshold that definesan amount of contrast difference that is sufficient to detect an edge.In an embodiment, thresholds used with Canny's algorithm can beadaptive. For instance, thresholds for areas of an image that have highinstances of edges (e.g., carpet, a table, or other textured surface)can be increased. This can reduce a number of false edges detected insuch areas. Likewise, thresholds for areas of the image that haverelatively low instances of edges can be decreased to increase thechances of detecting edges that have relatively small contrastdifferences.

At block 406, lines are determined that correspond to the detectededges. Capture application 110 can make use of detected edges toconstruct corresponding lines. In an implementation, lines aredetermined by applying a linear Hough transform. Lines can be identifiedto correct for imperfections that occur in detection of edges, such asincomplete edges, wavy edges, and so forth. This step can also involvescoring lines according to a scoring scheme and selectively discardinglines based on scoring criteria. For instance, isolated lines, linesthat are uncertain, and lines determined not to form suitablequadrangles can be discarded or ignored.

A variety of techniques to score and select lines are contemplated. Inan embodiment, a scoring refinement can be applied to a configurablenumber of top-scoring lines that are determined by application of theHough transform. In particular, each of the top scoring lines can bere-scored and re-sorted by scanning an area surrounding a line to findedges having a similar orientation. The new score for a particular lineis proportional to the number of similarly oriented edges found in thescanned area.

In an embodiment, the area scanned for a particular line can be adjustedbased upon the orientation of the line. For instance, for a line that issubstantially horizontal, the scanned area can include a configurablenumber of pixels above/below pixels of the line. Similarly, for a linethat is substantially vertical, the scanned area can include aconfigurable number of pixels to the left/right of pixels of the line.

At block 408, possible quadrangles are derived using the detected edgesand lines. In particular, a capture application 110 of a computingdevice 102 can make use of the edges detected at block 404 and linesdetermined at block 406 to construct possible quadrangles. In otherwords, various lines can be combined to form quadrangles, where thelines are detected from similarly-oriented edges along a particulardirection. Combining the lines to form quadrangles can occur in avariety of ways. In an example, detected lines can be processed first tofind possible quadrangles based upon finding lines that form oppositesides of a quadrangle. Detected lines can be processed again to findpossible quadrangles based upon lines that form quadrangle corners.Unlike some previous techniques, corners can correspond to irregular orskewed quadrangles, as well as to corners that form substantially rightangles.

At block 410, a quadrangle is selected for scanning. For instance, usingthe set of possible quadrangles constructed in block 408, the captureapplication 110 can be configured to automatically select a quadrangleaccording to scoring criteria. This step can involve identifyingmeaningful combinations of lines based upon a scoring scheme anddiscarding combinations that are considered un-meaningful. The scoringscheme can account for various criteria including location in the image,relative size, content contained within and outside of quadrangles, andso forth.

The scoring scheme can be employed by the capture application 110 toselect the likely quadrangle from among the possible quadrangles. Inother words, the capture application 110 can score the possiblequadrangles using the criteria to make a likely or approximatelybest-guess as to the intended subject of the captured image. Forexample, a large quadrangle near the center of the captured image islikely to be an intended subject and can be selected by the captureapplication 110 based upon the scoring. In contrast, a small quadranglelocated away from the center and having little or no variations in coloris likely not a meaningful quadrangle and can be discarded.

Various heuristics can be employed in order to find an optimalquadrangle. In an embodiment, the scoring-scheme can compute an initialscore for a particular quadrangle and then, optionally, modify theinitial score to account for various characteristics of the quadranglethat can increase or decrease the initial score. For example, an initialscore can be computed based upon a relative size of a quadrangle. Oneway an initial score based on relative size can be computed is bydividing the quadrangle's area by the image's area and taking a squareroot of this value. Optionally, various characteristics of quadranglescan be considered to modify the initial score. For instance, the initialscore can be modified with enhancements that increase the score and/orpenalties that reduce the score.

In one particular example, the initial score can be multiplied by, orotherwise adjusted by, various penalty factors to account for “unwanted”quadrangle characteristics. For instance, penalty factors can beconfigured as multipliers in a range between 0 and 1. A variety ofdifferent penalties and corresponding penalty factors are contemplated.For example, a penalty can be applied when two lines of a quadranglecorner extend past the corner. The penalty factor can be proportional tohow far the lines extend past the corner.

Another penalty can be based upon the angle formed between two lines atthe corner of a quadrangle. A penalty factor in this case can beproportional to the angle's difference from a right angle. Other examplepenalties can be assessed for quadrangles that extend past borders ofthe image, quadrangles that are significantly skewed relative to theborders of the image, and/or quadrangles that are positioned far awayfrom the center of the image or otherwise miss-aligned.

The initial score for a quadrangle can also be multiplied by, orotherwise adjusted by, various enhancement factors to account for“desired” quadrangle characteristics. For instance, example enhancementscan be applied for quadrangles located near the center of the image,that are substantially aligned with the image, that are well-formed, andso forth. It is noted that enhancements and penalties described hereincan be used individually and/or in combination to implement ascoring-scheme to select quadrangles.

At block 412, a determination is made as to whether the selection of thequadrangle is successful. For example, the capture application 110 candetermine when a suitable quadrangle has been selected. In some cases,the automatic selection of quadrangles at blocks 404-410 can fail todetect suitable quadrangles. For instance, capture application 110 candetermine that none of the possible quadrangles meet a defined scoringthreshold. In this case, the selection is determined to be unsuccessfuland semi-automatic correction can be initiated.

In another example, one or more quadrangles automatically selected atblock 410 can be presented via a user interface 112 of a computingdevice 102 for approval by the user. The user may then provide input toapprove or disapprove of the presented quadrangles. In this example, thedetermination at block 412 can be made according to the input providedby the user. If the user approves, then the selection is consideredsuccessful. If the user disapproves, the selection is determined to beunsuccessful and semi-automatic correction can be initiated.

If the selection is unsuccessful in the above scenarios, then at block414, semi-automatic correction is employed based upon user input toselect a quadrangle. Semi-automatic correction enables a user to provideinput to modify the automatic selection that can be made by the captureapplication 110. For example, one or more quadrangles automaticallyselected at block 410 can be presented via a user interface 112 of acomputing device 102. The presentation may make use of indicators 120 toshow the quadrangle boundaries. In an implementation, at least some ofthe indicators 120 are selectable by touch or other suitable input tomodify a corresponding quadrangle. Capture application 110 can detectinteraction with the indicators 120 and cause correspondingmodifications to quadrangles. For instance, a user may interact with theindicators 120 to make modifications such as resizing a quadrangle byselecting and dragging corner points (e.g., vertices), dragging to movethe quadrangle to a different location, rotating the quadrangle, and soforth.

In addition or alternatively, a user may define a custom quadrangle byselecting one or more corners through interaction with the userinterface 112. Again, this interaction can be by touch or other suitableinput. The capture application 110 can be configured to automaticallyderive a corresponding quadrangle responsive to user interaction toselect one corner using the techniques described herein. If the user isstill unsatisfied with the quadrangle, the user can select anothercorner and the capture application 110 can use both selected corners toderive a corresponding quadrangle. The process can be repeated a thirdtime by selection of a third corner. If the user is still unsatisfiedand a fourth corner is selected, capture application 110 can output aquadrangle whose vertexes correspond to the four selected corners. Inthis way, a user is able to provide successive hints to adjust aquadrangle that is automatically selected by the capture application110.

The user may also operate a select and drag tool of the user interface112 to identify an area for a custom quadrangle. In FIG. 1, for example,interaction of a user to select the illustration of a person in theviewfinder 114 of the user interface 112 is depicted. Following thesemi-automatic correction, method 400 proceeds to block 416.

At block 416, processing occurs to un-warp the perspective of theselected quadrangle. This step can occur both when the quadrangleselection is determined to be successful at block 412 and followingsemi-automatic correction at block 414. In general, un-warping isperformed to produce an un-warped image corresponding to the selectedquadrangle that is rectified to a rectangle. For instance, the captureapplication 110 can determine the perspective based upon angles andratios of the selected quadrangle. The capture application 110 can alsocrop the captured image to correspond to the selected quadrangle.Further, the capture application 110 can correct the perspective byrotating, resizing portions, and otherwise making corrections to accountfor the perspective distortions.

At block 418, visual enhancements are applied to the un-warped image.Various enhancements can be applied by the capture application 110 tothe un-warped image as discussed previously in relation to the foregoingfigures.

FIG. 5 illustrates various components of an example device 500 that canbe implemented as any type of portable and/or computer device asdescribed with reference to FIGS. 1 and 2 to implement embodiments ofcamera-based scanning. Device 500 includes communication devices 502that enable wired and/or wireless communication of device data 504(e.g., received data, data that is being received, data scheduled forbroadcast, data packets of the data, etc.). The device data 504 or otherdevice content can include configuration settings of the device, mediacontent stored on the device, and/or information associated with a userof the device. Media content stored on device 500 can include any typeof audio, video, and/or image data. Device 500 includes one or more datainputs 506 via which any type of data, media content, and/or inputs canbe received, such as user-selectable inputs, messages, music, televisionmedia content, recorded video content, and any other type of audio,video, and/or image data received from any content and/or data source.

Device 500 also includes communication interfaces 508 that can beimplemented as any one or more of a serial and/or parallel interface, awireless interface, any type of network interface, a modem, and as anyother type of communication interface. The communication interfaces 508provide a connection and/or communication links between device 500 and acommunication network by which other electronic, computing, andcommunication devices communicate data with device 500.

Device 500 includes one or more processors 510 (e.g., any ofmicroprocessors, controllers, and the like) which process variouscomputer-executable instructions to control the operation of device 500and to implement embodiments of camera-based scanning. Alternatively orin addition, device 500 can be implemented with any one or combinationof hardware, firmware, or fixed logic circuitry that is implemented inconnection with processing and control circuits which are generallyidentified at 512. Although not shown, device 500 can include a systembus or data transfer system that couples the various components withinthe device. A system bus can include any one or combination of differentbus structures, such as a memory bus or memory controller, a peripheralbus, a universal serial bus, and/or a processor or local bus thatutilizes any of a variety of bus architectures.

Device 500 also includes computer-readable media 514, such as one ormore memory components, examples of which include random access memory(RAM), non-volatile memory (e.g., any one or more of a read-only memory(ROM), flash memory, EPROM, EEPROM, etc.), and a disk storage device. Adisk storage device may be implemented as any type of magnetic oroptical storage device, such as a hard disk drive, a recordable and/orrewriteable compact disc (CD), any type of a digital versatile disc(DVD), and the like. Device 500 can also include a mass storage mediadevice 516.

Computer-readable media 514 provides data storage mechanisms to storethe device data 504, as well as various device applications 518 and anyother types of information and/or data related to operational aspects ofdevice 500. For example, an operating system 520 can be maintained as acomputer application with the computer-readable media 514 and executedon processors 510. The device applications 518 can include a devicemanager (e.g., a control application, software application, signalprocessing and control module, code that is native to a particulardevice, a hardware abstraction layer for a particular device, etc.). Thedevice applications 518 also include any system components or modules toimplement embodiments of camera-based scanning. In this example, thedevice applications 518 include a capture application 522 and an inputdriver 524 that are shown as software modules and/or computerapplications. Alternatively or in addition, the capture application 522and the input driver 524 can be implemented as hardware, software,firmware, or any combination thereof.

Device 500 also includes an audio and/or video input-output system 526that provides audio data to an audio system 528 and/or provides videodata to a display system 530. The audio system 528 and/or the displaysystem 530 can include any devices that process, display, and/orotherwise render audio, video, and image data. Such devices can includeat least a camera 532 to enable capturing video and images. Videosignals and audio signals can be communicated from device 500 to anaudio device and/or to a display device via an RF (radio frequency)link, S-video link, composite video link, component video link, DVI(digital video interface), analog audio connection, or other similarcommunication link. In an embodiment, the audio system 528 and/or thedisplay system 530 are implemented as external components to device 500.Alternatively, the audio system 528 and/or the display system 530 areimplemented as integrated components of example device 500. Likewise,the camera 532 can be implemented as an external or internal componentof the device 500.

Although embodiments of camera-based scanning have been described inlanguage specific to features and/or methods, it is to be understoodthat the subject of the appended claims is not necessarily limited tothe specific features or methods described. Rather, the specificfeatures and methods are disclosed as example implementations ofcamera-based scanning.

1. A computing device comprising: a memory and a processor configured toexecute instructions in the memory to implement a capture application,the capture application configured to: automatically select one or morequadrangular objects in a captured image for scanning; determine thatthe automatic selection of the one or more quadrangular objects issuccessful based on the one or more quadrangular objects having anassociated score that exceeds a predefined threshold; and create one ormore scanned documents from portions of the captured image correspondingto the selected one or more quadrangular objects, the one or morescanned documents including corrections to the portions for perspectivedistortion of the one or more quadrangular objects in the capturedimage.
 2. A computing device as recited in claim 1, wherein the captureapplication is further configured to automatically select the one ormore quadrangular objects by at least: detecting edges in the capturedimage based upon visual differences; and determining the one or morequadrangular objects as combinations of the detected edges.
 3. Acomputing device as recited in claim 2, wherein the capture applicationis further configured to: employ Canny's algorithm to detect the edgesand a Hough transform to form potential quadrangles from the detectededges; and apply a scoring scheme to the potential quadrangles todetermine the one or more quadrangular objects and identify theassociated score of the one or more quadrangular objects.
 4. A computingdevice as recited in claim 1, wherein the capture application is furtherconfigured to expose a user interface having a portion to display thecaptured image and another portion to concurrently display the one ormore scanned documents created from the captured image.
 5. A computingdevice as recited in claim 4, wherein the user interface is configuredto: present indicators within the captured image to identify theselected one or more quadrangular objects; enable user interaction withthe indicators to make a user modification to the selected one or morequadrangular objects; and responsive to the user interaction with theindicators, update and display the one or more scanned documents createdin accordance with the user modification.
 6. A computing device asrecited in claim 1, wherein the capture application is furtherconfigured to: create the one or more scanned documents using theautomatic selection when the automatic selection is successful; and inan event that the automatic selection is unsuccessful, create the one ormore scanned documents using quadrangular objects in the captured imageselected through a semi-automatic correction.
 7. A computing device asrecited in claim 1, wherein the capture application is furtherconfigured to, in an event that the automatic selection is unsuccessful,employ semi-automatic correction by at least: receiving a user input toselect a location in the captured image; and automatically deriving acorresponding quadrangle based upon the user input, the selectedlocation being one corner of the corresponding quadrangle.
 8. Acomputing device as recited in claim 1, wherein the capture applicationis further configured to apply one or more visual enhancements toimprove an appearance of the one or more scanned documents.
 9. Acomputing device as recited in claim 1, wherein the corrections to theportions for perspective distortions comprise rectification of each ofthe portions of the captured image to correspond to a rectangle.
 10. Amethod comprising: automatically selecting a portion of a captured imagethat corresponds to a rectangular object in the captured image based onthe portion having an associated score that exceeds a predefined scoringthreshold; creating a scanned document from the portion of the capturedimage; initiating a display of a user interface configured to present asplit screen having the captured image on one side of the split screenand the scanned document created from the captured image on another sideof the split screen; enabling user interaction with the captured imagevia the one side of the split screen to modify the portion of thecaptured image; and updating the scanned document displayed on the otherside of the split screen to reflect a modification caused by the userinteraction.
 11. A method as recited in claim 10, wherein the image iscaptured via a camera of a computing device.
 12. A method as recited inclaim 10, further comprising: detecting edges in the captured image;determining lines corresponding to the detected edges; derivingpotential quadrangles by combining the detected edges and lines; andusing the derived potential quadrangles for the automatic selection. 13.A method as recited in claim 12, wherein detecting the edges comprisesapplying an algorithm to find sharp changes of brightness correspondingto the edges.
 14. Computer-readable media having stored thereoncomputer-executable instructions that, if executed by a computingdevice, initiate the computing device to: identify at least onequadrangle in a captured image based upon a detection of one or morepotential quadrangles in the captured image, the at least one quadranglecorresponding to a rectangular object in the captured image; conduct anautomatic selection of the at least one quadrangle for scanning from thecaptured image; determine that the automatic selection is successfulbased on the selected at least one quadrangle having an associated scorethat exceeds a predetermined threshold; and based on the automaticselection being successful, process the captured image to produce ascanned document corresponding to the selected at least one quadrangle.15. Computer-readable media as recited in claim 14, wherein thecomputer-executable instructions, if executed by the computing device,cause the computing device to perform the detection of the one or morepotential quadrangles by at least: detecting edges in the capturedimage; determining lines corresponding to the detected edges; andderiving the one or more potential quadrangles by combining the detectededges and the determined lines.
 16. Computer-readable media as recitedin claim 15, wherein detecting the edges comprises applying an algorithmto find sharp changes of brightness corresponding to the edges. 17.Computer-readable media as recited in claim 15, wherein determininglines corresponding to the detected edges comprises applying a lineartransform to correct imperfections in the detected edges. 18.Computer-readable media as recited in claim 14, wherein the automaticselection is conducted by at least applying a scoring scheme to the oneor more potential quadrangles detected in the captured image thataccounts for at least a location of the one or more potentialquadrangles in the captured image and relative size of the one or morepotential quadrangles in the captured image.
 19. Computer-readable mediaas recited in claim 14, wherein the captured image is processed toproduce the scanned document by at least: cropping the captured image toprovide a cropped image that corresponds to the at least one quadrangle;un-warping the cropped image to correct for perspective distortions; andapplying one or more visual enhancements to the cropped image. 20.Computer-readable media as recited in claim 14, wherein thecomputer-executable instructions, if executed by the computing device,cause the computing device to output a user interface having a portionto display the captured image and another portion to concurrentlydisplay the scanned document produced by processing the captured image.